Hadoop on Windows with Eclipse

1. Introduction
2. Prerequistes
3. Set Environment Variables
4. Setup SSH daemon
5. Download hadoop and place it in the home directory.
6. Unpack hadoop
7. Configure Hadoop
8. Format the namenode
9. Setup hadoop plugin
10. Start the cluster
11. Setup hadoop location
12. Upload data
13. Create and run a test project.

Setup SSH daemon

Both Hadoop scripts and Eclipse plug-in need password-less SSH to operate. This section describes how to set it up in the Cygwin environment.

Configure ssh daemon

1. Open the Cygwin command prompt.
2. Execute the following command:
   
   ssh-host-config
3. When asked if privilege separation should be used, answer no.
4. When asked if sshd should be installed as a service, answer yes.
5. When asked about the value of CYGWIN environment variable, enter ntsec.
6. Here is an example session of this command. Note that the input typed by the user is shown in pink and output from the system is shown in gray.
   
   
   Example of using ssh-host-config

Start SSH daemon

1. Find My Computer icon either on your desktop or in the start-up menu, right-click on it and select Manage from the context menu.
2. Open Services and Applications in the left-hand panel then select the Services item.
3. Find the CYGWIN sshd item in the main section and right-click on it.
4. Select Start from the context menu.
   

A small window should pop-up indicating the progress of the service start-up. After that window disappears the status of CYGWIN sshd service should change to Started.

Setup authorization keys

Eclipse plug-in and Hadoop scripts require ssh authentication to be performed through authorization keys rather than passwords. The following steps describe how authorization keys are set up.

1. Open cygwin command prompt
2. Execute the following command to generate keys
   
   ssh-keygen
3. When prompted for filenames and pass phrases press ENTER to accept default values.
4. After the command has finished generating keys, enter the following command to change into your .ssh directory:
   cd ~/.ssh
5. Check if the keys were indeed generated by executing the following command:
   ls -l
   
   You should see two files id_rsa.pub and id_rsa with recent creation dates. These files contain authorization keys.
6. To register the new authorization keys enter the following command (note the sharply-angled double brackets -- they are very important):
   cat id_rsa.pub >> authorized_keys
   

   

   Setting up authorization keys

   
   

7. Now check if the keys were set up correctly by executing the following command:
   ssh localhost
   Since it is a new ssh installation, you will be warned that authenticity of the host could not be established and will be asked whether you really want to connect. Answer yes and press ENTER. You should see the Cygwin prompt again, which means that you have successfully connected.
8. Now execute the command again:
   ssh localhost
   
   This time you should not be prompted for anything.

 

Continue